Designing and Evaluating a Semantic Annotation Scheme for Compound Nouns

There is no standard set of semantic relations for classifying noun-noun compounds. This paper describes the development of a new annotation scheme which fulfils a number of desirable criteria. A rigorous dual-annotator experiment indicates that reasonably good agreement can be achieved but that the task remains a very difficult one. Analysis of the annotators' disagreements suggests which categories are most problematic and identifies specific cases for which the annotation guidelines could be further refined. Nonetheless there is a very long tail of disagreement patterns which render infeasible the production of fully exhaustive guidelines.